{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "pGUYKbJNWNgj"
      },
      "source": [
        "##### Copyright 2020 The TensorFlow Authors."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "cellView": "form",
        "id": "1PzPJglSWgnW"
      },
      "outputs": [],
      "source": [
        "#@title Licensed under the Apache License, Version 2.0\n",
        "# you may not use this file except in compliance with the License.\n",
        "# You may obtain a copy of the License at\n",
        "#\n",
        "# https://www.apache.org/licenses/LICENSE-2.0\n",
        "#\n",
        "# Unless required by applicable law or agreed to in writing, software\n",
        "# distributed under the License is distributed on an \"AS IS\" BASIS,\n",
        "# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
        "# See the License for the specific language governing permissions and\n",
        "# limitations under the License."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "b5P4BEg1XYd5"
      },
      "source": [
        "# TensorFlowアドオンオプティマイザ：ConditionalGradient\n",
        "\n",
        "<table class=\"tfo-notebook-buttons\" align=\"left\">\n",
        "  <td><a target=\"_blank\" href=\"https://www.tensorflow.org/addons/tutorials/optimizers_conditionalgradient\"><img src=\"https://www.tensorflow.org/images/tf_logo_32px.png\"> TensorFlow.orgで表示</a></td>\n",
        "  <td><a target=\"_blank\" href=\"https://colab.research.google.com/github/tensorflow/docs-l10n/blob/master/site/ja/addons/tutorials/optimizers_conditionalgradient.ipynb\"><img src=\"https://www.tensorflow.org/images/colab_logo_32px.png\"> Google Colab で実行</a></td>\n",
        "  <td><a target=\"_blank\" href=\"https://github.com/tensorflow/docs-l10n/blob/master/site/ja/addons/tutorials/optimizers_conditionalgradient.ipynb\"><img src=\"https://www.tensorflow.org/images/GitHub-Mark-32px.png\">GitHub でソースを表示{</a></td>\n",
        "  <td><a href=\"https://storage.googleapis.com/tensorflow_docs/docs-l10n/site/ja/addons/tutorials/optimizers_conditionalgradient.ipynb\"><img src=\"https://www.tensorflow.org/images/download_logo_32px.png\">ノートブックをダウンロード/a0}</a></td>\n",
        "</table>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Faj8luWnYNSG"
      },
      "source": [
        "# 概要\n",
        "\n",
        "このノートブックでは、アドオンパッケージのConditional Gradientオプティマイザの使用方法を紹介します。"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "MrDjqjY6YRYM"
      },
      "source": [
        "# ConditionalGradient\n",
        "\n",
        "> 根本的な正則化の効果を出すために、ニューラルネットワークのパラメーターを制約することがトレーニングに有益であることが示されています。多くの場合、パラメーターはソフトペナルティ（制約充足を保証しない）または投影操作（計算コストが高い）によって制約されますが、Conditional Gradient（CG）オプティマイザは、費用のかかる投影ステップを必要とせずに、制約を厳密に適用します。これは、制約内のオブジェクトの線形近似を最小化することによって機能します。このノートブックでは、MNISTデータセットに対してCGオプティマイザを使用してフロベニウスノルム制約を適用する方法を紹介します。CGは、tensorflow APIとして利用可能になりました。オプティマイザの詳細は、https://arxiv.org/pdf/1803.06453.pdfを参照してください。\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "dooBaYGLYYnn"
      },
      "source": [
        "## セットアップ"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "2sCyoNXlgGbk"
      },
      "outputs": [],
      "source": [
        "!pip install -U tensorflow-addons"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "qYo0FkL4O7io"
      },
      "outputs": [],
      "source": [
        "import tensorflow as tf\n",
        "import tensorflow_addons as tfa\n",
        "from matplotlib import pyplot as plt"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "kR0PnjrIirpJ"
      },
      "outputs": [],
      "source": [
        "# Hyperparameters\n",
        "batch_size=64\n",
        "epochs=10"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "-x0WBp-IYz7x"
      },
      "source": [
        "# モデルの構築"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "4KzMDUT0i1QE"
      },
      "outputs": [],
      "source": [
        "model_1 = tf.keras.Sequential([\n",
        "    tf.keras.layers.Dense(64, input_shape=(784,), activation='relu', name='dense_1'),\n",
        "    tf.keras.layers.Dense(64, activation='relu', name='dense_2'),\n",
        "    tf.keras.layers.Dense(10, activation='softmax', name='predictions'),\n",
        "])"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "XGADNG3-Y7aa"
      },
      "source": [
        "# データの準備"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "d6a-kbM_i1b2"
      },
      "outputs": [],
      "source": [
        "# Load MNIST dataset as NumPy arrays\n",
        "dataset = {}\n",
        "num_validation = 10000\n",
        "(x_train, y_train), (x_test, y_test) = tf.keras.datasets.mnist.load_data()\n",
        "\n",
        "# Preprocess the data\n",
        "x_train = x_train.reshape(-1, 784).astype('float32') / 255\n",
        "x_test = x_test.reshape(-1, 784).astype('float32') / 255"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "sOlB-WqjZp1Y"
      },
      "source": [
        "# カスタムコールバック関数の定義"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "8LCmRXUgZqyV"
      },
      "outputs": [],
      "source": [
        "def frobenius_norm(m):\n",
        "    \"\"\"This function is to calculate the frobenius norm of the matrix of all\n",
        "    layer's weight.\n",
        "  \n",
        "    Args:\n",
        "        m: is a list of weights param for each layers.\n",
        "    \"\"\"\n",
        "    total_reduce_sum = 0\n",
        "    for i in range(len(m)):\n",
        "        total_reduce_sum = total_reduce_sum + tf.math.reduce_sum(m[i]**2)\n",
        "    norm = total_reduce_sum**0.5\n",
        "    return norm"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "udSvzKm4Z5Zr"
      },
      "outputs": [],
      "source": [
        "CG_frobenius_norm_of_weight = []\n",
        "CG_get_weight_norm = tf.keras.callbacks.LambdaCallback(\n",
        "    on_epoch_end=lambda batch, logs: CG_frobenius_norm_of_weight.append(\n",
        "        frobenius_norm(model_1.trainable_weights).numpy()))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "qfhE1DfwZC1i"
      },
      "source": [
        "# トレーニングと評価：オプティマイザとしてCGを使用\n",
        "\n",
        "一般的なkerasオプティマイザを新しいtfaオプティマイザに置き換えるだけです。 "
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "6-AMaOYEi1kK"
      },
      "outputs": [],
      "source": [
        "# Compile the model\n",
        "model_1.compile(\n",
        "    optimizer=tfa.optimizers.ConditionalGradient(\n",
        "        learning_rate=0.99949, lambda_=203),  # Utilize TFA optimizer\n",
        "    loss=tf.keras.losses.SparseCategoricalCrossentropy(),\n",
        "    metrics=['accuracy'])\n",
        "\n",
        "history_cg = model_1.fit(\n",
        "    x_train,\n",
        "    y_train,\n",
        "    batch_size=batch_size,\n",
        "    validation_data=(x_test, y_test),\n",
        "    epochs=epochs,\n",
        "    callbacks=[CG_get_weight_norm])"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "8OJp4So9bYYR"
      },
      "source": [
        "# トレーニングと評価：オプティマイザとしてSGDを使用"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "SuizUueqn449"
      },
      "outputs": [],
      "source": [
        "model_2 = tf.keras.Sequential([\n",
        "    tf.keras.layers.Dense(64, input_shape=(784,), activation='relu', name='dense_1'),\n",
        "    tf.keras.layers.Dense(64, activation='relu', name='dense_2'),\n",
        "    tf.keras.layers.Dense(10, activation='softmax', name='predictions'),\n",
        "])"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "V8QC3xCwbfNl"
      },
      "outputs": [],
      "source": [
        "SGD_frobenius_norm_of_weight = []\n",
        "SGD_get_weight_norm = tf.keras.callbacks.LambdaCallback(\n",
        "    on_epoch_end=lambda batch, logs: SGD_frobenius_norm_of_weight.append(\n",
        "        frobenius_norm(model_2.trainable_weights).numpy()))"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "9BNi4yXGcDlg"
      },
      "outputs": [],
      "source": [
        "# Compile the model\n",
        "model_2.compile(\n",
        "    optimizer=tf.keras.optimizers.SGD(0.01),  # Utilize SGD optimizer\n",
        "    loss=tf.keras.losses.SparseCategoricalCrossentropy(),\n",
        "    metrics=['accuracy'])\n",
        "\n",
        "history_sgd = model_2.fit(\n",
        "    x_train,\n",
        "    y_train,\n",
        "    batch_size=batch_size,\n",
        "    validation_data=(x_test, y_test),\n",
        "    epochs=epochs,\n",
        "    callbacks=[SGD_get_weight_norm])"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "1Myw0FVcd_Z9"
      },
      "source": [
        "# 重みのフロベニウスノルム：CGとSGDの比較"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "0tJYQBRt-ZUl"
      },
      "source": [
        "現在のCGオプティマイザの実装はフロベニウスノルムに基づいており、フロベニウスノルムをターゲット関数の正則化機能と見なしています。ここでは、CGオプティマイザの正規化された効果を、フロベニウスノルム正則化機能のないSGDオプティマイザと比較します。"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "Ewf17MW1cJVI"
      },
      "outputs": [],
      "source": [
        "plt.plot(\n",
        "    CG_frobenius_norm_of_weight,\n",
        "    color='r',\n",
        "    label='CG_frobenius_norm_of_weights')\n",
        "plt.plot(\n",
        "    SGD_frobenius_norm_of_weight,\n",
        "    color='b',\n",
        "    label='SGD_frobenius_norm_of_weights')\n",
        "plt.xlabel('Epoch')\n",
        "plt.ylabel('Frobenius norm of weights')\n",
        "plt.legend(loc=1)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "JGtutiXuoZyx"
      },
      "source": [
        "# トレーニングと検証の精度：CGとSGDの比較\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "s-SNIr10o2va"
      },
      "outputs": [],
      "source": [
        "plt.plot(history_cg.history['accuracy'], color='r', label='CG_train')\n",
        "plt.plot(history_cg.history['val_accuracy'], color='g', label='CG_test')\n",
        "plt.plot(history_sgd.history['accuracy'], color='pink', label='SGD_train')\n",
        "plt.plot(history_sgd.history['val_accuracy'], color='b', label='SGD_test')\n",
        "plt.xlabel('Epoch')\n",
        "plt.ylabel('Accuracy')\n",
        "plt.legend(loc=4)"
      ]
    }
  ],
  "metadata": {
    "accelerator": "GPU",
    "colab": {
      "collapsed_sections": [],
      "name": "optimizers_conditionalgradient.ipynb",
      "toc_visible": true
    },
    "kernelspec": {
      "display_name": "Python 3",
      "name": "python3"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}
